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THREE-DIMENSIONAL DATA PROCESSING METHOD, THREE-DIMENSIONAL 
DATA PROCESSING PROGRAM, AND THREE-DIMENSIONAL DATA 

PROCESSING SYSTEM 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0001] The present invention relates to a three- 

dimensional image generating technique which obtains data 
necessary for generating a three-dimensional image of a real 
existing object by means of CG (computer graphics), and more 
specifically, a technique which adds a bump map to a three- 
dimensional shape model shown by a polygon mesh. 

2. Description of Related Art 

[0002] Conventionally, to reproduce a real existing object 
by means of CG, a method in which an image of the real 
existing object taken by a camera is pasted on the portions 
of a three-dimensional shape model has been used. In such a 
method, in order to cope with CG of various observing points, 
a plurality of images taken from a plurality of directions 
are prepared, and for reproduction, appropriate images are 
selected and pasted (refer to Japanese Patent Application 
Laid-Open No. 2000-348213). 

[0003] However, by this method, images of an object to be 
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reproduced by means of CG, illuminated under illumination 
conditions different from those when the object is imaged, 
cannot be obtained. 

[0004] Furthermore, images taken by a camera originally 
include shades and highlights, so that an object pasted with 
the taken images becomes unnatural when it is provided with 
further shades and highlights. 

[0005] Therefore, a method is employed in which a change 
in reflectance due to a light source direction or an 
observing direction is expressed as a reflection function, a 
numerical expression model is introduced for the reflection 
function, and a surface reflectance properties is indicated 
by reflection model constants (refer to "Object Shape and 
Reflectance Modeling from Observation:" SIGGRAPH Computer 
Graphics Proceedings, Annual Conference Series, pp379-387, 
1997) . 

[0006] This method realizes reproduction of natural shades 
and highlights in optional illumination conditions and 
observing directions by use of comparatively slight surface 
reflectance properties information by holding reflection 
characteristics of colored portions provided on the surface 
of a three-dimensional shape model as reflection model 
constants, and when reproducing by substituting the 
constants into a reflection model function. 

[0007] However, the abovementioned reflection model 
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expresses light reflection and diffusion reflecting the 
surface structure at the micro level. Therefore, it cannot 
reflect fine shades due to unevenness that can be viewed, 
such as a stone surface, loosely woven cloth, and leathery 
crepe, etc. Of course, such unevenness can be expressed by a 
polygon mesh, however, in this case, the number of vertices 
and the number of polygons become enormous and result in a 
high operation processing burden, and this is not realistic. 
[0008] On the other hand, in CG drawing, for drawing the 
abovementioned fine shades, a method called bump mapping is 
employed. 

[0009] Generally, bump mapping is not concerned with shape 
drawing, but is used for shading by influencing normal 
directions at fine portions. 

SUMMARY OF THE INVENTION 

[0010] An object of the invention is to provide a three- 
dimensional data processing technique which realizes natural 
CG reproduction of a real existing object by a data amount 
and a calculation amount as small as possible. 
[0011] In order to achieve the abovementioned object, 

according to one aspect of the three-dimensional data 
processing method, program, and system of the invention, 
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first data showing at least one of a surface shape and taken 
images of a real existing object is acquired, and a bump map 
for generating a three-dimensional image of the object is 
generated based on the first data. 

[0012] The characteristics of the invention will be 

clarified by the following detailed description of 
embodiments with reference to drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0013] Fig. 1 is a block diagram showing the structure of 
a three-dimensional data processing system of Embodiment 1 
of the invention . 

[0014] Fig. 2 is an explanatory view showing a flow of 

data in the three-dimensional data processing system of 
Embodiment 1 . 

[0015] Fig. 3 is an explanatory view of a bump map 

generated in the three-dimensional data processing system of 
Embodiment 1 . 

[0016] Fig. 4 is a flowchart showing operations of the 

three-dimensional data processing system of Embodiment 1. 
[0017] Fig. 5 is a block diagram showing the structure of 
a three-dimensional data processing system of Embodiment 2 
of the invention. 
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[0018] Fig. 6 is an explanatory view showing a flow of 

data in the three-dimensional data processing system of 
Embodiment 2 . 

[0019] Fig. 7 is a schematic view showing an acquirement 
example of image data in the three-dimensional data 
processing system of Embodiment 2. 

[0020] Fig. 8 is a flowchart showing operations of the 

three-dimensional data processing system of Embodiment 2. 
[0021] Fig. 9 is a flowchart showing a surface reflectance 
properties estimating operation in the three-dimensional 
data processing system of Embodiment 2. 

[0022] Fig. 10 is a conceptual diagram showing surface 

reflectance properties estimation in the three-dimensional 
data processing system of Embodiment 2. 

[0023] Fig. 11 is an explanatory view of Roll 0 (|>(N). 
[0024] Fig. 12 is a flowchart showing a surface 

reflectance properties estimating operation in a three- 
dimensional data processing system of Embodiment 3 of the 
invention . 

[0025] Fig. 13 is a block diagram showing the structure of 
a three-dimensional data processing system of Embodiment 4 
of the invention. 

[0026] Fig. 14 is an explanatory view showing a flow of 
data in the three-dimensional data processing system of 
Embodiment 4 . 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0027] Hereinafter, the three-dimensional data processing 
technique of the invention is described with reference to 
the accompanying drawings. 

[0028] (Embodiment 1) 

Fig. 1 shows the structure of a three-dimensional data 
processing system 100 of Embodiment 1 of the invention, and 
Fig. 2 shows a flow of data among components of the three- 
dimensional data processing system 100 of Fig. 1. 
[0029] The three-dimensional data processing system 100 of 
this embodiment comprises a shape acquiring section 130, a 
polygon mesh generating section 135, a bump map generating 
section 140, a model data storing section 150, a CG 
generating section 160, a CG display section 170, a control 
section 180, and a memory 190. Herein, as shown in Fig. 1, 
the respective sections of the three-dimensional data 
processing system are structured in a manner enabling them 
to exchange data via, for example, a bus interface, etc. 
[0030] This three-dimensional data processing system 100 

may be structured as one unit (computer) , or may be 
structured by combining separate units. In this embodiment, 
description is given by assuming that the system is 
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structured as one unit, however, the three-dimensional data 
processing system may be composed of, for example, a main 
unit comprising the shape acquiring section 130, the polygon 
mesh generating section 135, the bump map generating section 
140, the model data storing section 150, the control section 
180, and the memory 190, and a separate unit comprising the 
CG generating section 160 and the CG display section 170. 
[0031] The shape acquiring section 130 acquires highly 

accurate three-dimensional shape data of an object O. The 
highly accurate three-dimensional shape data shows the 
surface shape of the object O, and shows three-dimensional 
coordinates of the respective surface portions of the object 
O. As this shape acquiring section 130, a three-dimensional 
shape acquiring apparatus such as a rangefinder or a three- 
dimensional scanner, etc., can be used. 

[0032] Furthermore, the shape acquiring section 130 

outputs a three-dimensional shape model shown by the 
acquired highly accurate three-dimensional shape data to the 
polygon mesh generating section 135 and the bump map 
generating section 140. 

[0033] It is also possible that the shape acquiring 

section 130 is structured as a unit separate from the main 
unit, and highly accurate three-dimensional shape data 
acquired by this separate unit is stored in a recording 
medium such as a semiconductor memory, a magnetic disk, or 
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an optical disk to make it possible to input (read) the data 
into the main unit. It is still also possible that the main 
unit has an input function of highly accurate three- 
dimensional shape data from such a storage medium and the 
shape acquiring section 130. 

[0034] The polygon mesh generating section 135 generates a 
polygon mesh based on the highly accurate three-dimensional 
shape data input from the shape acquiring section 130. The 
polygon mesh shows the object's shape by using an assembly 
of line segments connected in a network pattern. Generation 
of the polygon mesh based on the highly accurate three- 
dimensional shape data is carried out according to a 
calculation method generally known. The polygon mesh 
generating section 135 outputs the generated polygon mesh to 
the bump map generating section 140 and the model data 
storing section 150. 

[0035] Herein, the polygon mesh generating section 135 can 
control the level of complexity (the number of vertices and 
the number of polygons) of the polygon mesh to be generated, 
and an operator of this system can specify the level of 
complexity of the polygon mesh to be generated by 
considering the necessary accuracy, drawing speed and data 
size. Furthermore, the level of complexity specified in this 
embodiment shall be lower (that is, the shape is simpler) 
than the level of complexity of the highly accurate three- 
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dimensional shape data. 

[0036] The bump map generating section 140 generates a 

bump map on the basis of the highly accurate three- 
dimensional shape data input from the shape acquiring 
section 130 and the polygon mesh input from the polygon mesh 
generating section 135. The bump map is a distribution map 
including normal directions changed with fixed ordinality, 
and is used for fine shading. By ordinality differences on 
the bump map, it becomes possible to express different 
surface conditions such as a leathery surface, a metallic 
surface, and a ceramic surface, etc. The bump map generating 
section 140 outputs the generated bump map to the model data 
storing section 150. Detailed operations of the bump map 
generating section 140 are described later. 

[0037] The model data storing section 150 stores the 

polygon mesh generated by the polygon mesh generating 
section 135 and the bump map generated by the bump map 
generating section 140. The model data storing section 150 
can use various storage media, and the abovementioned memory 
190 can be simultaneously used as the model data storing 
section 150. The model data storing section 150 can output 
the stored polygon mesh and bump map to the CG generating 
section 160. 

[0038] The CG generating section 160 generates three- 

dimensional image data of the object 0 based on the polygon 
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mesh and the bump map input from the model data storing 
section 150, in various observing points and illumination 
environments according to specification of an operator. 
Concretely, shape drawing by using the polygon mesh and 
shading by using the bump map are carried out. 
[0039] Normal directions are necessary for this shading. 
On the other hand, the bump map generated by the bump map 
generating section 140 is positional displacement data. 
Normal fluctuations (relative normal directions) are 
obtained by differentiating the positional displacements 
along the polygon surfaces, and by calculating the 
differences between the normal fluctuations and the normal 
directions estimated from the polygon mesh, normal 
directions of respective texels of the bump map are obtained. 
[0040] The CG display section 170 displays a three- 

dimensional image generated by the CG generating section 160. 
As this CG display section 170, various display apparatuses 
such as a CRT, an LCD, etc., can be used. Furthermore, as 
the CG display section 170, a printing unit such as a 
printer that can print image data may also be used. 
[0041] The control section 180 comprises a processor such 
as a CPU or an MPU, and controls operations of the 
respective sections by using operation programs stored in 
the memory 190. 

[0042] The memory 190 includes a nonvolatile memory such 
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data necessary for processing of the three-dimensional 
processing system 100, and a volatile memory such as a RAM 
which temporarily stores image data and necessary control 
programs . 

[0043] Next, operations of the three-dimensional data 

processing system 100 structured as mentioned above (program 
operations under control of the control section 180) are 
described in detail with reference to the flowchart of Fig. 
4. In Fig. 4 (and flowcharts of other figures), steps are 
abbreviated to "S." 

[0044] (Step 1: Shape acquiring step) 

The shape acquiring section 130 acquires highly accurate 
three-dimensional shape data by measuring the shape of the 
object O. The highly accurate three-dimensional shape data 
shows at least three-dimensional coordinates of measuring 
points optionally set at respective portions on the 
surfaceof the object O. Herein, the measuring points whose 
three-dimensional coordinates are shown are sufficiently 
large in number in comparison with the number of vertices of 
a polygon mesh to be generated by the polygon mesh 
generating section 135 later. 

[0045] The shape acquiring section 130 outputs the 

acquired highly accurate three-dimensional shape data to the 
polygon mesh generating section 135 and the bump map 
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generating section 140. 

[0046] (Step 2, Step 3: Polygon mesh generating steps) 

The polygon mesh generating section 135 converts the 
highly accurate three-dimensional shape data input from the 
shape acquiring section 130 into a polygon mesh. At this 
point, a polygon mesh simplified in shape complexity up to a 
level specified by an operator is generated. 

[0047] As a detailed polygon mesh generating method 

(algorithm) herein, the method introduced in Hoops H. et al., 
"Mesh Optimization, " Computer Graphics Proceedings 
SIGGRAPH '93 can be used. However, the polygon mesh 
generating method is not limited to this method, and other 
various methods can be used. 

[0048] For each vertex of this simplified polygon mesh, 

bump texture coordinates that specify the position of bump 
map pasting are provided. Then, the polygon mesh provided 
with the bump texture coordinates is output to the bump map 
generating section 140 and stored in the model data storing 
section 150. 

[0049] (Step 4: Bump map generating step) 

The bump map generating section 140 sets texels adapted 
to the level of complexity of the polygon mesh input from 
the polygon mesh generating section 135. Furthermore, the 
bump map generating section 140 compares the highly accurate 
three-dimensional shape data input from the shape acquiring 
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section 130 and the polygon mesh input from the polygon mesh 
generating section 135, calculates differences between these 
as displacements for respective texels, and outputs the 
displacements as a bump map to the model data storing 
section 150. The model data storing section 150 stores the 
bump map . 

[0050] Herein, Fig. 3 is an explanatory view showing the 
abovementioned displacements. The solid line shows a piece 
of the shape shown by the highly accurate three-dimensional 
shape data, and the dotted line shows a piece of the shape 
shown by the polygon mesh. Texels exist on the polygon mesh, 
and from the respective texel positions, arrows indicating 
the displacements extend. The lengths of the arrows of the 
respective texels are output to and recorded on the model 
data storing section 150 as a bump map. Regarding the signs 
of the displacements, the sign is positive (meaning convex) 
when the corresponding arrow points to the outside of the 
polygon mesh (points up in Fig. 3) while the sign is 
negative (meaning concave) when the corresponding arrow 
points to the inside (points down in Fig. 3) . 

[0051] Thereby, the polygon mesh and the bump map 

necessary for CG reproduction of the object are stored in 
the model data storing section 150. 
[0052] (Step 5: CG generating step) 

For CG reproduction of the object O, the CG generating 
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section 160 calculates the normal directions, illumination 
(incidence) directions, and image-taking (observing) 
directions of the respective texels based on layout 
information of the image-taking apparatus and the object and 
the illumination distribution in a virtual space required in 
the CG generating section 160 and the three-dimensional 
shape model (polygon mesh and bump map) stored in the model 
data storing section 150, and then calculates observation 
luminances. By drawing the observation luminances as display 
colors at display positions on the generated image, a three- 
dimensional image can be displayed on the CG display section 
170. 

[0053] When the normal directions are calculated, by 

taking the bump map into account in addition to the normals 
calculated from the polygon mesh, fine shades are also 
reproduced within the polygon. 

[0054] As in this embodiment, based on highly accurate 

three-dimensional shape data, a simplified polygon mesh and 
a bump map which corresponds it are prepared separately, and 
these polygon mesh and bump map are handled as a pair of 
shape data for CG reproduction, etc., whereby a reproduced 
image (three-dimensional image) of an object having fine 
unevenness and shades can be obtained with a small data 
amount and a small calculation amount in comparison with the 
case where CG reproduction, etc., are realized by directly 
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using the highly accurate three-dimensional shape data. 

[0055] (Embodiment 2) 

Fig. 5 shows the structure of a three-dimensional data 
processing system 200 of Embodiment 2 of the invention. Fig. 
6 shows a flow of data among components of the three- 
dimensional data processing system 200 shown in Fig. 5. 
[0056] The three-dimensional data processing system 200 of 
this embodiment comprises an image data acquiring section 
110, an image-taking parameter acquiring section 120, a 
shape acquiring section 130, a polygon mesh generating 
section 135, a surface reflectance properties estimating 
section 145, a model data storing section 150, a CG 
generating section 160, a CG display section 170, a control 
section 180, and a memory 190. 

[0057] Herein, as shown in Fig. 6, the respective sections 
of the three-dimensional data processing system 200 are 
structured in a manner enabling them to exchange data via, 
for example, a bus interface, etc. 

[0058] This three-dimensional data processing system 200 

may be structured as one unit (computer) , or may be composed 
of separate units. In this embodiment, description is given 
by assuming that the system is structured as one unit, 
however, it is also possible that the system is structured 
as a three-dimensional data processing system composed of a 
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main unit comprising the image data acguiring section 110, 
the image-taking parameter acguiring section 120, the shape 
acquiring section 130, the polygon mesh generating section 
135, the surface reflectance properties estimating section 
145, the model data storing section 150, the control section 
180, and the memory 190 and a separate unit comprising the 
CG generating section 160 and the CG display section 170. 
[0059] The image data acquiring section 110 acquires taken 
image data of an object O. In this embodiment, the image 
data acquiring section 110 comprises an image-taking 
apparatus 112 and a light source 114. Namely, by taking 
images of the object O illuminated by light from the light 
source 114 by the image-taking apparatus 112, image data of 
the object O is acquired. 

[0060] The image data acquiring section 110 acquires a 

plurality of images that are different in observing (image- 
taking) direction, light source (illuminating) direction, or 
posture (direction) of the object O. For example, as shown 
in Fig. 7, the object O is placed on a turntable S that is 
rotatable around an axis vertical to a surface on which the 
object O is placed, and the image-taking apparatus 112 and 
the light source 114 are changed in angle (that is, image- 
taking direction) to the object 0 from one direction 
independently or in synchronization with each other, whereby 
image-taking from a plurality of directions is realized. 
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[0061] The image-taking apparatus 112 and the light source 
114 are fixed on, for example, a tripod, and their image- 
taking angles can be changed by changing the height of the 
tripod. However, it is also possible that the image-taking 
apparatus 112 and the light source 114 are connected to 
another moving mechanism that is not shown, and the moving 
mechanism is controlled by the control section 180, whereby 
the angles of the image-taking apparatus 112 and the light 
source 114 are automatically changed for image-taking. 

[0062] As the image-taking apparatus 112, a digital still 
camera or a video camera can be used. The image-taking 
apparatus 112 produces image data composed of three R, G, 
and B color bands and outputs the data to the surface 
reflectance properties estimating section 145. As the light 
source 114, any of a point light source, a line light source, 
and a surface light source may be used. 

[0063] Furthermore, the light source 114 is not 

necessarily an exclusive-use light source, and by providing 
an illumination environment measuring mechanism, a light 
source such as normal indoor illumination or sunlight, etc., 
can be used as the light source 114. In this case, measured 
illumination environment information is defined as light 
source information. For example, an orbit camera can be used 
as the illumination environment measuring mechanism. 

[0064] Furthermore, in this embodiment, the object O is 
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placed on the turntable S that is uniaxially rotatable, 
however, it is also possible that the object 0 is supported 
on an apparatus having a plurality of rotation axes. With 
such a structure, even when the image-taking apparatus 112 
and the light source 114 are fixed at predetermined 
locations, image data on images of the object taken from a 
plurality of directions can be acquired. Furthermore, the 
system can also be structured so that the object 0 is fixed 
and the image thereof is taken while the position of the 
image-taking apparatus 112 or the light source 114 is 
changed, or image-taking is carried out while two or more 
parameters out of the position of image-taking apparatus 112, 
the position of the light source 114 and the posture of the 
object 0 are changed. 

[0065] Furthermore, in this embodiment, description is 

given on the assumption that the image data acquiring 
section 110 is a component of the main unit, however, it is 
also possible that the image data acquiring section 110 is 
structured as a separate unit. In this case, image data 
obtained through image-taking is stored in a recording 
medium such as a semiconductor memory, a magnetic disk, or 
an optical disk, etc., so that inputting (reading-in) of the 
data into the main unit is made possible. Furthermore, a 
structure is also possible in which the main unit has both 
the input function of the image data from such a recording 
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medium and the image data acquiring section 110. 
[0066] The image-taking parameter acquiring section 120 

acquires image-taking parameters for the image data in the 
image data acquiring section 110. In greater detail, the 
image-taking parameter acquiring section 120 acquires image- 
taking information including angles of view of the image- 
taking apparatus 112 and environment information in image- 
taking including the image-taking position (observing point) 
and luminance, position, and posture information of the 
object O and the light source 114. The image-taking 
parameter acquiring section 120 outputs acquired image- 
taking parameters to the surface reflectance properties 
estimating section 145. 

[0067] The image-taking parameters may be directly input 

by an operator (including a person who took images or an 
operator of the three-dimensional data processing system 
200) into the surface reflectance properties estimating 
section 145 via an input apparatus such as a keyboard. In 
this case, it is necessary that the abovement ioned image 
data acquiring section 110 records layout of the image- 
taking apparatus 112, the light source 114, and the object O, 
the angle of view of the image-taking apparatus 112, and 
luminance information of the light source 114 in image- 
taking for each image to be acquired. Furthermore, in a case 
where the image data acquiring section 110 is controlled by 
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the control section 180 and image-taking of the object O is 
automatically carried out, image-taking parameters for each 
image may be acquired by the control section 180. 
[0068] Furthermore, it is also possible that the image- 

taking parameter acquiring section 120 is not provided and 
image-taking parameters stored in a recording medium such as 
a semiconductor memory, a magnetic disk, or an optical disk 

are read-in as data. It is still also possible that the 

i 

reading-in function of image-taking parameters from such a 
recording medium and the image-taking parameter acquiring 
section 120 are simultaneously provided. 

[0069] In addition, a part of the image-taking parameters 
may be estimated by the control section 1.80 from image data 
acquired by the image data acquiring section 110 in place of 
being recorded when images are taken. In this case, markers 
fixed on the object O are picked up in images together with 
the object 0 when the images are taken. When the object O 
has characteristic points that serve in place of the markers, 
the characteristic points may be used as the markers. Then, 
image-taking parameters are estimated from the positions and 
colors of the markers on taken images. An image-taking 
parameter estimating method is introduced in Chapter 6 of 
"Computer Vision" (Corona Corporation, 1999), however, other 
estimating methods can also be used. 

[0070] The shape acquiring section 130 and the polygon 
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mesh generating section 135 have the same function as in 
Embodiment 1. Namely, the shape acquiring section 130 
acquires three-dimensional shape data of an object O by 
using a three-dimensional shape acquiring apparatus such as 
a rangefinder or a three-dimensional scanner, etc., and 
outputs the data to the polygon mesh generating section 135. 
Then, the polygon mesh generating section 135 generates a 
polygon mesh having a specified level of complexity based on 
the three-dimensional shape data input from the shape 
acquiring section 130 (or read-in from a recording medium) . 

[0071] In this embodiment, the shape acquiring section 130 
outputs the three-dimensional shape data to only the polygon 
mesh generating section 135. In this embodiment, the three- 
dimensional shape data is used only for polygon mesh 
generation, and is not directly used for bump generation. 
Therefore, the shape measuring points are smaller in number 
than in Embodiment 1, and accordingly, a lower-accuracy 
three-dimensional shape acquiring apparatus can be used. 

[0072] Furthermore, the shape acquiring section 130 may 

acquire three-dimensional shape data by means of a 
silhouette method from image data acquired by the image data 
acquiring section 110 (or read-in image data) . 

[0073] The polygon mesh generating section 135 outputs the 
generated polygon mesh to the surface reflectance properties 
estimating section 145 and the model data storing section 
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150. 

[0074] Herein, it is possible that the shape acquiring 

section 130 and the polygon mesh generating section 135 are 
formed as separate apparatuses and inputting of three- 
dimensional shape data or a polygon mesh stored in a 
recording medium such as a semiconductor memory, a magnetic 
disk, or an optical disk, etc., into the main unit is made 
possible. Furthermore, the system can be structured so as to 
have the input function from such a recording medium, the 
shape acquiring section 130, and the polygon mesh generating 
section 135. 

[0075] Based on the information input from the image data 
acquiring section 110, the image-taking parameter acquiring 
section 120, and the polygon mesh generating section 135, 
the surface reflectance properties estimating section (bump 
map generating part) 145 estimates surface reflectance 
properties information for each texel on a texture map to be 
pasted on the polygon mesh. Herein, surface reflectance 
properties mentioned in this embodiment includes a 
reflection model constant and a normal fluctuation of each 
texel. Detailed processing of the surface reflectance 
properties estimating section 145 is described later. 

[0076] The surface reflectance properties estimating 

section 145 outputs the estimated surface reflectance 
properties information to the model data storing section 150. 
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[0077] The model data storing section 150 stores the 

polygon mesh generated by the polygon mesh generating 
section 135 and the surface reflectance properties of the 
respective texels estimated by the surface reflectance 
properties estimating section 145. A texture map recording 
normal fluctuations of the surface reflectance properties 
serves as a bump map. 

[0078] Various storage media can be used as the model data 
storing section 150, or the abovementioned memory 190 can be 
simultaneously used as the data model storing section. The 
model data storing section 150 outputs the scored polygon 
mesh and information on surface reflectance properties to 
the CG generating section 160 . 

[0079] Furthermore, the functions of the CG generating 

section 160, the CG display section 170, the control section 
180, and the memory 190 are the same as those in Embodiment 
1. 

[0080] Next, operations of the three-dimensional data 

processing system 200 of this embodiment (program operations 
under control of the control section 180) are described with 
reference to the flowchart of Fig. 8. 
[0081] (Step 11: Image data acquiring step) 

First, in the image data acquiring section 110, as shown 
in Fig. 7, the object O is placed on the turntable S and 
illuminated by the light source 114. At this point, the 
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image-taking apparatus 112 is turned toward the object 0. 
Then, images of the object are taken from a plurality of 
angles by the image-taking apparatus 112 while turning the 
turntable S to acquire image data composed of three R, G, 
and B color bands. Thereby, image data of images with 
changes in posture of the object 0 and relative differences 
in image-taking direction can be acquired. Furthermore, by 
changing the angle of the image-taking apparatus 112 or the 
light source 114 with respect to the object O, more image 
data can be obtained. The image data acquiring section 110 
outputs these pieces of image data to the surface 
reflectance properties estimating section 145. 
[0082] (Step 12: Image-taking parameter acquiring step) 

In the image-taking parameter acquiring section 120, 
when image-taking is carried out, information on the image- 
taking apparatus 112 and the light source 114 and 
information on the positions and the postures of the image- 
taking apparatus 112 and the object O are acquired. Then, 
these pieces . of information are output to the surface 
reflectance properties estimating section 145. Furthermore, 
as mentioned above, a part of the image-taking parameters 
may be estimated from image data in place of being recorded 
when image-taking is carried out. 

[0083] . (Step 13: Surface shape acquiring step) 

The shape acquiring section 130 acquires three- 
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dimensional shape data by measuring the shape of the object 
0. This three-dimensional shape data shows at least three- 
dimensional coordinates of measuring points optionally set 
at respective portions on the surface of the object O. 
Herein, the measuring points whose three-dimensional 
coordinates are shown are sufficiently large in number in 
comparison with the number of vertices of a polygon mesh to 
be generated by the polygon mesh generating section ' 135 
later . 

[0084] The shape acquiring section 130 outputs the 

acquired three-dimensional shape data to the polygon mesh 
generating section 135. 

[0085] (Step 14, Step 15: Polygon mesh generating step) 

The polygon mesh generating section 135 converts the 
three-dimensional shape data input from the shape acquiring 
section 130 into a polygon mesh. At this point, a polygon 
mesh that has been simplified in shape complexity up to the 
level specified by an operator is generated. 

[0086] At the respective vertices of this simplified 

polygon mesh, texture coordinates that specify a pasting 
position of a texture map are provided. Then, the polygon 
mesh provided with the texture coordinates is output to the 
surface reflectance properties estimating section 145 and 
stored in the model data storing section 150. 

[0087] The texture map mentioned herein includes bump map 
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recording normal displacements of the respective texels and 
a reflection model constant map storing at least a part of 
the reflection model constants, and both of these have the 
same resolution and pasting position. 

[0088] (Step 16: Surface reflectance properties estimating 
step) 

In the surface reflectance properties estimating section 
145, the surface reflectance properties of the object O are 
estimated on the basis of the input various pieces of 
information, and the obtained reflection model constants are. 
stored in the model data storing section 150. Herein, the 
program operations of the surface reflectance properties 
estimating section 145 in this surface reflectance 
properties estimating step are described with reference to 
the flowchart of Fig. 9. 

[0089] The surface reflectance properties estimating 

section 145 arranges texels with the specified resolution on 
the texture map (Step 161) . This resolution is generally set 
to 2n x 2n, and desirably, set to be substantially 
equivalent to the occupation size of the image data pixels 
on the surface of the object O. However, the resolution is 
not limited to this. Then, the following processing is 
repeated for each texel (Step S162) . 

[0090] First, as shown in Fig. 10, for all pieces of image 
data that have been taken by the abovementioned image-taking 
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method and are different in posture of the object 0, image- 
taking direction and illumination direction, the normal 
direction vector N ba se, the light source direction vector L, 
the light source illuminance I c , the observing direction 
vector V, and the on-image luminance (reflection luminance) 
E c estimated from the polygon mesh in one texel are 
calculated, and a table of correspondence is created (Step 
S163) . 

[0091] Herein, the subscript "C" of the light source 

illuminance I and the on-image luminance E shows a color 
band, and generally, C = {R, G, B}. 

[0092] The normal direction, the light source direction, 
and the observing direction can be calculated from the 
image-taking parameters and the polygon mesh. Namely, from 
information on the postures and layout of the image-taking 
apparatus 112, the light source 114, and the object O and 
the polygon mesh at the time of image-taking, the normal 
direction, the observing direction, the light source 
direction, and the position on the image of each texel are 
calculated. Then, the luminance can be obtained from the 
image . 

[0093] Furthermore, in a. case where the on-image luminance 
is unknown since the texel is on the back surface of the 
image, behind another portion, or outside the image, it is 
excluded from the table of correspondence. 
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[0094] Next, this table of correspondence is substituted 
into the reflection model function BRDF parms (N; V, L) which 
is determined by considering the illumination distribution 

(Step 164). For example, when a Phong reflection model shown 
in Formula (1) is used as a reflection model function, the 
reflection model constant "parms" becomes: 
{Cd R , Cd G , Cd B , C s , n} 

[0095]. 

BRDFparms (N; V, L) = Cd c (N-L) + C s (V-L) n ...(1) 

Formula (1) is equivalent to Formula (2) indicated by 

tangential coordinates using N as the z axis. 

[0096] 

BRDF' parms (V , L' ) = Cd c L'z + C S (V, L' ) n ...(2) 

Herein, V and L' are expressions of V and L in a tangential 

coordinate system, and L' z is a z component of L' . 

[0097] Herein, when the normal on a texel is further 

defined as N tex and an operation Roller (N) to rotate the 
vector N shown in Fig. 11 around the axis of the tangential 
vector T of the polygon by 0 and around the axis of another 
tangential vector B orthogonal to the T axis by <|> is 
introduced, the relationship of Formula (3) is obtained. 

[0098] 

Ecj = I c BRDF parms (N tex = Rolled, (N base j ) ; V j , L j ) + B c j...(3) 
Herein, e shows an error, and the subscript j shows an image 
number . 
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[0099] Then, a combination of constants which makes the 

abovement ioned error optimum: 
{Cd R/ Cd G , Cd B , C s , n, 9, 

is determined, and reflection model constants of the texel: 
(Cd R/ Cd G , Cd B , C S / n) 

and the normal fluctuation (9, <|>) are defined, and these are 
stored in (recorded on) the model data storing section 150 
(Step 165) . 

[0100] For the abovement ioned optimization, various 

mathematical methods can be used. For example, in a case 
where the least-squares method is used, the evaluation 
function is defined as the sum of squares of the error e c j as 
shown in Formula (4), and a combination of the constants 
which minimizes this evaluation function are determined. 
[0101] 

f(Cd R ,Cd G ,Cd B ,Cs,n,^^) = X(^/ +£gj +£b-) • • ■ < 4 > 

j 

where, P denotes each image of the table of 
correspondence . 

[0102] The abovementioned evaluation function f handles 

all errors equivalently, however, it is also possible that 
an evaluation function set by considering the reliabilities 
of the respective values is set. 

[0103] Furthermore, in the above explanation, the light 

source is set at only one point in the V direction, however, 
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in a case where a plurality of light sources exist or a 
light source is not a point light source but has expansion 
on the celestial sphere, the light source distribution is 
defined as I C (L) and Formula (5) that is an integral 
expression of these light sources is applied in place of 
Formula ( 3 ) . 
[0104] 

B C J = J R I c (Lj)BRDF P arms(N tex = Roll,, (N has J),Vj\Lj)dLj + e c j . . . ( 5 ) 

where, R denotes Region including light sources 
distributed. 

[0105] Thereby, complete model data necessary for CG 

reproduction of the object O is prepared in the model data 
storing section 150. The normal fluctuations obtained for 
each texel form a bump map. 
[0106] (Step 17: CG generating step) 

For reproducing the object O by means of CG, the CG 
generating section 160 calculates the normal directions, the 
illumination (incidence) directions, and the image-taking 
(observing) directions of each texel based on the layout 
information of the image-taking apparatus and the object and 
the illumination distribution in a virtual space required in 
the CG generating section 160, and the polygon mesh and the 
surface reflectance properties data (including the bump map) 
stored in the model data storing section 150, and 
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substitutes these into the reflection model function to 
calculate the observation luminances. By drawing the 
observation luminances as display colors at display 
positions on a generated image, a three-dimensional image 
can be displayed on the CG display section 170. 
[0107] When calculating the normal directions, by 

considering the bump map in addition to the normals 
calculated from the polygon mesh, fine shades can be 
reproduced in a polygon. 

[0108] Furthermore, although the case where the image data 
is composed of three colors of R, G, and B has been 
described in this embodiment, the image data may be composed 
of a single color, other colors, spectral information, or 
image of polarized light as long as it can describe a 
corresponding reflection model function . 

[0109] Furthermore, although the case using a Phong 

reflection model as the reflection model function has been 
described in the abovementioned embodiment, another 
reflection model function may be used. For example, a 
reflection model function of a Lambert model, a Blinn model, 
or a Torrance-Sparrow model, etc., can also be used as long 
as a reflectance is provided from the relationship between 
one or a plurality of reflection model constants and the 
normal direction, the light source direction, and the 
observing direction . 
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[0110] Moreover, the reflection model function that can be 
used in this embodiment is a bidirectional reflection 
distribution function which provides a reflectance by using 
at least the light source direction and the observing 
direction in the tangential coordinate system as arguments, 
and is a numerical expression having reflection model 
constants as elements. 

[0111] The reflection model function that can be used in 
this embodiment is a dichroic reflection model expressed by 
a linear combination of diffuse reflection components and 
specular reflection components. In this case, reflection 
model constants may be roughly divided into two groups of 
diffuse reflection model constants concerning diffuse 
reflection components and specular reflection model 
constants concerning specular reflection components. 

[0112] Then, the reflection model constants and the normal 
directions or normal fluctuations constituting the bump map 
are set to a combination of values that most accurately 
reproduce measured luminance changes determined from image 
data when the light source directions (illumination 
directions) and the observing directions (image-taking 
directions) determined from image-taking parameters are 
applied to the reflection model function. 

[0113] In an estimating method of the surface reflectance 
properties estimating section 145, reflection model 
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constants and normal displacements are not always uniquely 
determined in all texels. However, such a problem can be 
solved by complement from reflection constants of a texel 
near the texel having constants that cannot be determined or 
substitution of typical values. 

[0114] Although the normal fluctuations of texels are 

stored in the model data storing section 150. Further, it 
is possible that a bump map in the abovement ioned embodiment, 
it is also possible that normal directions are determined 
from differences between the normal fluctuations and normal 
directions estimated from the polygon mesh, and are stored 
as a bump map in the data storing section 150, or 
displacements that are differences between the positions 
estimated from the polygon mesh and the positions on the 
three-dimensional shape are determined and stored as a bump 
map in the data storing section 150. To convert a normal 
fluctuation into a displacement, the fact that the 
displacement z and the normal fluctuation (0, p) satisfy the 
relationship of the following Formula (6) is utilized. 
[0115] 

tan (9) = — , tan (((>) = — ... (6) 
dB dT 

From Formula (6), a restriction of Formula (7) is obtained. 
[0116] 
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rot grad z = rot 



(tamO)} dtan(G , dtan(<j» =o 



tan (<(>) 



dT dB 



Herein, T and B are unit vectors of tangential line 
directions of the polygon mesh orthogonal to each other. 
[0117] When estimating the normal fluctuations, the 

estimation accuracy can be improved by considering the 
restriction of Formula (7). 

[0118] In the abovementioned embodiment, in the surface 

reflectance properties estimating section 145, a case where 
the surface reflectance properties are estimated at the same 
resolution as that in recording on the model data storing 
section 150 has been described, however, it is also possible 
that the surface reflectance properties are estimated at 
higher resolutions • in the surface reflectance properties 
estimating section 145 and the resolutions are converted 
into a specified resolution by being averaged among texels 
when the attributes are stored in the model data storing 
section 150. Further, it is possible that noise influence 
is suppressed by carrying out processing such as applying a 
low pass filter in the tangential line directions when 
storing in the model data storing section 150. 

[0119] In this embodiment, surface reflectance properties 
information including a bump map is estimated and generated 
not from shape data but from shades of image data of an 
object, so that high accuracy and high measuring point 
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density are not required for shape acquirement. Therefore, a 
three-dimensional data processinq apparatus can be realized 
by using a comparatively simple three-dimensional shape 
acquiring apparatus in the shape acquiring section 130. 

[0120] (Embodiment 3) 

Next, a three-dimensional data processing system of 
Embodiment 3 of the invention is described. The system 
structure of this embodiment is similar to that of 
Embodiment 2 shown in Fig. 5 and Fig. 6, however, operations 
of the surface reflectance properties estimating section 145 
are different from those in Embodiment 2, so that the 
operations of the surface reflectance properties estimating 
section 145 are described herein. 

[0121] In this embodiment, the surface reflectance 

properties estimating section 145 estimates surface 
reflectance properties information of each texel on the 
basis of information input from the image data acquiring 
section 110, the image-taking parameter acquiring section 
120, and the polygon mesh generating section 135. Herein, 
the surface reflectance properties information mentioned in 
this embodiment includes a surface reflectance properties 
index and a normal fluctuation of each texel. The surface 
reflectance properties estimating section 145 outputs 
estimated surface reflectance properties information to the 
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data model storing section 150. 

[0122] The program operations of the surface reflectance 
properties estimating section 145 are described with 
reference to the flowchart of Fig. 12. Operations of Step 
161, Step 162, and Step 163 are the same as in Embodiment 2. 
[0123] In Step 164', a correspondence table prepared in 

Step 163 is compared with samples in a surface reflectance 
properties list in order. The surface reflectance properties 
list is a reflectance table TABLE' k(V, L' ) indicating a 
series of reflectances with respect to the light source 
directions L' and observing directions V in the tangential 
coordinate systems using normal directions as z axes in 
various samples k. 

[0124] When a converting formula into a tangential 

coordinate system defining a normal as N is assumed to be V 
= trans (N; V), this reflectance table TABLE' k(V, L' ) can 
be expressed as Formula (8). 
[0125] 

TABLE k (N ; V, L) = TABLE' k ( V = trans (N; V), L' 

= trans (N; L) . . . (8) 
Furthermore, the normal on a texel is defined as N teX / and an 
operation Roller (N) that rotates the vector N around the 
axis of the tangential line vector T of the polygon by 0 and 
around the axis of another tangential vector B orthogonal to 
the T axis by <|> is introduced, and regarding an image 
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luminance when the surface reflectance properties of the 
texel is sample number k, the relationship shown by Formula 
(9) is obtained. 
[0126] 

E c j = Ic TABLE k (N te x = Roller (Nbasej ) ; V j , L j ) + e c j ..-(9) 

where, 8 shows an error, and the subscript j shows an 
image number. 

[0127] Then, a sample number k to correct the error to an 
optimum value is determined, and is stored in the model data 
storing section 150 as the surface reflectance properties 
index k and the normal fluctuation (0, <|)) . 

[0128] Thereby, complete model data necessary for 

reproducing the object by means of CG is collected in the 
model data storing section 150. 

[0129] In this embodiment, the surface reflectance 

properties is obtained not as a reflection model constant 
but as an index indicating one within the list, so that when 
the surface reflectance properties of the object are limited 
to several types, stable estimation can be carried out. 

[0130] (Embodiment 4) 

Fig. 13 shows the structure of a three-dimensional data 
processing system 300 of Embodiment 4 of the invention, and 
Fig. 14 shows a flow of information among components of the 
three-dimensional data processing system 300. 
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[0131] The three-dimensional data processing system 300 of 
this embodiment comprises a shape acquiring section 130, a 
polygon mesh input section 136, a bump map generating 
section 140, a model data storing section 150, a CG 
generating section 160, a CG display section 170, a control 
section 180, and a memory 190. 

[ 0132 ] Furthermore, the functions of the shape acquiring 

section 130, the bump map generating section 140, the model 
data storing section 150, the CG generating section 160, the 
CG display section 170, the control section 180, and the 
memory 190 are the same as those in Embodiment 1. Operations 
of the system are almost the same as those described in the 
flowchart of Fig. 4 in Embodiment 1. However, in the present 
embodiment, in place of the polygon mesh generating section 
135 of Embodiment 1, the polygon mesh input section 136 is 
provided. Step 2 in the flowchart of Fig. 4 serves as a 
polygon mesh input step. 

[0133] The polygon mesh input section 136 acquires a 

polygon mesh to be used for three-dimensional data 
generation of an object O. Concretely, in Step 2 of the 
flowchart of Fig. 4, the polygon mesh input section reads-in 
information on the polygon mesh that has been processed by 
another apparatus and stored in a recording medium such as a 
semiconductor memory, magnetic disk, or optical disk, or 
receives information on the polygon mesh through wireless or 
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wired communications lines from the another apparatus that 
processes the polygon mesh. 

[0134] The polygon mesh input section 136 can controls the 
level of complexity of the polygon mesh. When the input 
polygon mesh has complexity different from a required level 
of complexity, the polygon mesh input section 136 increases 
or lowers the level of complexity of the polygon mesh so 
that the level agrees with the required level of complexity. 
[0135] Furthermore, when the polygon mesh is not provided 
with the bump texture coordinates, the polygon mesh input 
section 136 provides to the polygon mesh. 

[0136] In a case where reference coordinate systems are 

different between the highly accurate three-dimensional 
shape data input from the shape acquiring section 130 and 
the polygon mesh, the polygon mesh input section 136 
converts the coordinates of the polygon mesh or the highly 
accurate three-dimensional shape data so that the coordinate 
systems agree with each other. The polygon mesh thus 
optimized is output from the bump map generating section 140 
and stored in the model data storing section 150. Then, as 
in Embodiment 1, a three-dimensional image is generated and 
displayed by using the polygon mesh and the bump map stored 
in the model data storing section 150. 

[0137] In this embodiment, it becomes possible that a 

common basic polygon mesh for expressing, for example, faces 
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of persons is prepared, and the shapes of individual faces 
of the persons are generated as different bump maps on the 
basic polygon mesh. Thereby, when face data of a plurality 
of persons is held, only one polygon mesh is held, and 
therefore, the data amount can be saved. 

[0138] Even when special processing such as morphing of a 
certain person' s face into another person' s face is carried 
out, only the bump map is processed without changing the 
polygon mesh, and this simplifies the processing. 
[0139] Furthermore, in each embodiment mentioned above, as 
a method of obtaining data (three-dimensional shape data, 
image data, polygon mesh data, etc.) necessary for 
generating a bump map, any other method can be used in 
addition to the methods described in each embodiment. 

Furthermore, it is also possible that a higher accurate 
three-dimensional image is obtained by combining the method 
of generating a bump map based on surface shape data and the 
method of generating a bump map based on taken image data 
described in the respective embodiments mentioned above. 
[0140] As described above, according to the embodiments 

mentioned above, based on surface shape data or taken image 
data of a real existing obj ect , a bump map expressing fine 
unevenness of the object surface can be generated, so that a 
three-dimensional image of the object having fine shades 
based on the surface unevenness can be reproduced with a 
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small data amount and calculation amount. 

[0141] While preferred embodiments have been described, it 
is to be understood that modification and variation of the 
present invention may be made without departing from scope 
of the following claims. 



